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The Fourteenth Annual 
Western Regional Conference on 

Testing Problems 

The fourteenth annual meeting of the Western Regional Conference 
on Testing Problems was convened at 9:15 a,m., May 7, 1965, at 
the Hilton Inn, San Francisco International Airport, by Mr. Robert 
E. Lambert. Director of Advisory Services for the Western Oifice 
of the Educational Testing Service. 

Mr, Lambert: I'm Bob Lambert, Director of Advisory Services for 
the Western Ofiice of ETS. It is my distinct pleasure to welcome 
you to the Fourteenth Conference and to introduce your Chairman, 
Dr. Theodore L. Reller. 

Dr. Reller s background resum^, which was compiled eithek by a 
dutiful wife or an adoring secretory, is frightfully impressive. 
Because he does not need the addition of another stone to the 
ahready impressive cairn of his past adiievements and honors, I 
shall give but a few facts from his vita. He received his Baccalau- 
reate and Masters Degree at the University of Pennsylvania. He 
then moved to Yale where he earned the Ph.D. He has been a 
teacher and a supervising principal of public schools. He served as 
an instructor to the Professor at the University of Pennsylvania. 
As you know, he is presently Dean of the School of Education at 
the University of California. 

Without further delay I entrust the day's convention fortunes to 
the capable hands of Dr. Reller* 

CHAiRMiiN Reuxr: Thank you Mr. Lambert 

Today we are gathered to look at the question, at one of the 
more important questiOiis-perhap* the most important-of educa- 
tion in our and other societies, the question of Change* We live in 
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a period that is characterized by something approaching revolution 
if it not be revolution. We see this in the increase, of population. 
We see it in the increase of nations, in knowledge, in technology. 
In educational expectation we have seen a revolution so great that, 
as a matter of fact, many people are concerned whether or not it 
will be possible to live somewhat in accord with this revolution in 
expectations. And if we should not there may be unfortunate 
consequences. 

When I think of this matter I recaU a meeting at which a speaker 
for the Soviet Union traced the very enormous developments in 
education in the Soviet society over a period of forty years and 
indicated the extent to which actually the building of the Soviet 
society had been an educational operation. He printed a very glow- 
ing picture of forty or a little more than forty years of development. 
And when he finished, the Chairman of the meeting, who was the 
Minister of Education in Nigeria said, "But it took you forty years. 
Who's interested in anyone who moves as slowly as that? This lias 
no meaning for our society. We must do this in five years.** This 
indicates something of the ferment, the spirit of revolution that 
runs through our societies. 

I think it is important that we recognize that change is not always 
even. You get change in one level and in one area but at different 
times. Also I think we need to note that there is considerable resist- 
ance to change in any one situation. Both of these things, I think, 
characterize educational developments in our own society. 

With reference to Change, the matter of appraisal or evaluation 
is extremely important for it is particularly, perhaps peculiarly, re- 
lated to a society or to an educational system during a period of 
great change, for appraisal and evaluation can either facilitate or 
retard. I recall members of faculties in Indian univet^ities who say 
that they cannot really deal with anything that has occurred in recent 
years because it is not covered by the examinations and the students 
simply revolt if you deal with anything that is modem because they 
know it doesn't relate to the examinations they will be taking. 



■ This morning we are going to turn first to a couple of areas in which 
there has been most substantial ferment and change. We are veiy fortu- 
nate today in having a group of outstanding speakers to give us some 
picture of^ the changes that are occurring. Our first speaker will deal with 
some of the problems related to the area of mathematics. 

I tiiink I need take veiy little time to comment regarding his compe- 
tence for this assignment. He certainly is as outstanding a person as could 
have been found for this particular area. Having completed his under- 
graduate work in the science of engineering at the University of Cincin- 
nati he went on to complete his doctorate in mathematics at Ohio State. 
His thesis v/as "^The Nature of Definition in High School Geometry.** He 
has been with ETS since 1955 as Head of the Mathematics Department 
and has been veiy deeply engaged in the development of appraisal and 
the approaches to testing during this period of time. 

He is noted for his relaxed, yet passionate approach to the field of 
maliiematics. He has been very mudi interested in the measurement of 
the insightful and non-routine thinking of young people. To ask the 
searching question that at once challenges and stimulates is the main 
objective of his work. He believes all students should be encouraged to 
explore and to discover mathematical concepts on their own, should have 
the opportunity to solve novel problems jmd should acquire notions of 
how a mathematician thinks. He will be leading us in our thinking this 
morning dealing with the subject, The Winds rf Change in Mathematics 
Education with a Discussion of some of the Stronger Gusts." It gives me 
a great deal of pleasure to present to you Dr. Sheldon Myers. Dr. Myers. 

The Winds of Change in Mathematics 
Education with a Discussion of Some 

of the Stronger Gusts 

SHELDON S. MYERS 

This paper will attempt to provide an overview and some historical 
perspective on the current situation in mathematics education. The 
major characteristics of contemporary mathematics grades K-12 will 
be described and some of the curriculum projects will be discussed* 
One of the first series of national grants for the improvement of 
education occurred some time ago. The government was a monardhy. 
The receiving institution was a university with its library located in 
a pleasant, sub-tropical climate at the delta of a giant river. One of 
the professors involved was able to reorganize the loose, informal, 
intuitive, non-rigorous mathematics of his day into a polished, rig- 
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orous deductive system. The curricular materials that he wrote and 
tried out on adult students were published as a handwritten Greek 
textbook of 13 chapters. The professor was Euclid; the textbook was 
The Elements of Geometry; the source of funds was one of the 
Ptolomies of Egypt; the institution was the University of Alexandria 
at the delta of tiie Nile; and the period was about 300 B.C. to 
273B.C. 

Euclid received his pre-service training in the acad'roy of Plato 
in Athens. Aristotle received his training in Plato's academy and 
P.^ato in turn received his training from a private tutor, Socrates. 
Although Euclid patterned his deductive structure consisting of un- 
defined terms, axioms, postulates, and theorems on the system of 
logic described by Aristotle in the Organon, he was the first to 
organize a branch of mathematics into an almost completely rigor- 
ous and deductive system. Euclid's Elements stands as a timeless 
intellectual monument to one of the greatest mathematicians in his- 
tory. In style, elegance, and uniqueness, the Elements is comparable 
to the statues of Michelangelo, the paintings of Rembrandt, the 
fugues of Bach, the quartettes of Beethoven, and the plays of 
Shakespeare. 

The Elements was not written as an immediate response to prac- 
tical Jiuman needs. Rather, they were written in a leisurely academic 
environment as an exercise of the intellect. Most of the geometric 
facts of the Elements were known several thousand years earlier to 
the Egyptians and Babylonians. The annual flooding of the Nile 
wiped out boundary lines between farms facing the river. These 
were re-established each year by surveying methods utilizing the 
empirical geometry known to that day. A base line, above the high 
water mark, ran parallel to the river bank. The boundary lines 
between farms intersected the base line at right angles. Egyptian 
surveyors called Tiarpedonaptae,** or rope stretchers, would resurvey 
the boundary lines after the annual flood. For this purpose, they 
used a rope knotted in three parts in the ratio 3 to 4 to 3. By 
stretching the rope into a right triangle, at the appropriate points 
on the base line, the washed out boundary lines could be redrawn 
perpendicular to the base line and parallel to each other. 

The Babylonians not only knew the geometric relationship later 
called *'the Pythagorean Theorem** but they also defined the degree 
unit for measuring arcs and angles. Their astronomy was highly 
advanced and they knew Aat the earth required about 360 days to 
m^e a trip around the sun. The portion of arc traversed in one day 
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was therefore taken as a basic unit equal to 1 degree. The base of 
60 of the Babylonian system of numeration ("sexagesimals") was 
probably also derived from the number 360. 

Most of the Greek forbears of Euclid in geometry knew the sub- 
ject from an intuitive, empirical point of view. From Pythagoras 
who established a school at the Greek colony of Croton in southern 
Italy, to Thales who established the Ionian school of geometry after 
wandering as a merchant through the Near East, to Eudoxus who 
developed the theory of proportions and incommensurables which 
Euclid largely used, most of the geometry prior to Euclid was a 
collection of isolated facts with only an occasional, fragmentary, 
deductive chain of theorems. 

Euclid's great contribution was to organize this body of knowl- 
edge into a self-contained, autonoir - deductive system. This 
system began with primitive assumptions consisting of axioms, pos- 
tulates and terms and a sequence of theorems, each theorem being 
deductively proven by means of previously proven theorems and/or 
assumptions. The first six books of Euclid consist of the theorems 
now known as plane geometry. Books VII through X consi.^^t of 
number theory and the geometric equivalent of elementary algebra. 
Books XI through XIII consist of theorems now known as solid 
geometry. Although Euclid's Elements had some logical flaws, such 
as attempting to define all terms and proving the contrapositive of 
a theorem already proven, it stands as the first example of a postu- 
lational or mathematical system in history. Next to the Bible, the 
Elements is the most widely printed and studied book in the Western 
world. 

In Euclid, mathematics had swung from an intuitive, empirical, 
and applied science to a rigorous, autonomous, and pure mathe- 
matical system. Although an obscure woik of Euclid, known as the 
Optics, indicates that Euclid was not unaware of applications, the 
impact of the Elements through the centuries has been on rigor for 
its own sake, rather than on the applied aspect of mathematics. 

The history of mathematics education has been marked by swings 
between informal, applied mathematics and rigorous, abstract maflie- 
matics. After the destruction of the University and Library at Alex- 
andria, the Elements became lost to the Western world and was 
known for a period only to the Arabs in Arabic translations. Our 
first Latin version of the Elements was made available by Adelard 
of Bath about 1120 A.D. who translated the Arabic version available 
in Spain. Our first English version was made available by Billings- 
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ley in 1370 by translating the Latin version of Adelard. From this 
time on, the rigorous Euclidean approach dominated the British 
approach to geometry. The French, on the other hand, gradually 
evolved a more informal, intuitive geometry. By the middle of the 
eighteenth century the French were making extensive use of rota- 
tions, reflections and symmetry in an informal, intuitive, non- 
rigorous way. 

The great French mathematician, Legendre, attempted to reform 
the loose, informal teaching of geometry in France by publishing 
in 1794 the first mathematically rigorous text of geometry based on 
the requirements of modem education. This text departed signifi- 
cantly from Euclid's sequence. 

Euclid's Elements came to this country via England in the form 
of translations by Todhunter and others. The text of Legendre came 
to this country via translations by Charles Davies, professor of 
mathematics at West Point. The Davies translation of Legendre 
became the most widely used textbook of geometry in the United 
States until the geometty textbook of Wentworth around 1900. 

Aside from the few universities in this country where geometry 
was taught via Todhunter or Legendre, most of the colonial mathe- 
matics in this country at the academy and early college level was of 
the applied kind. Colonial arithmetic dealt with reckoning tables 
of interest ;^.nd measures. Most arithmetic problems were business 
problems. Geometry below the college level was mostly mensura- 
tional. 

I once spent a week at the University of Michigan examining 
the Karpinski collection of mathematics textbooks published in the 
United States before 1850. I also examined student notebooks in 
mathematics from colonial days. In one notebook, I found the 
methods of calculating the volumes of 14 different types of hay- 
stacks. In others I saw beautiful, colored maps drawn in ink repre- 
senting the solution of surveying problems. The trigonometry of 
those days was taught chiefly for the purpose of navigating the early 
Yankee sailing vessels or for surveying new land on the expanding 
fronHer. 

The curriculum of the early American academy was packed with 
a great variety of practical courses. These academies prevailed until 
gradually replaced by the comprehensive high school in the latter 
part of the nineteenth century. By the end of the eighteenth cen- 
tury the subjects of geometry and algebra had moved downward 
from sophomore to freshman subjects in college and, in the early 



MATHEMATICS EDUCATION 7 



nineteenth century, because requirements for admission. Thus alge- 
bra and geometry became traditionally secondary school subjects. 

With the spread of the geometry texts of Todhunter, Legendre, 
and Wentworth, the teaching of geometry in this country became 
relatively formal and the predominantly applied aspect of colonial 
mathematics decreased greatly. Arithmetic was tau^t by formal 
rules, such as the rule of three, aliquoting, etc., and cyphering was 
taught like calisdienics. Cyphering matches rivaled spelling matches 
as social events in Darke County, Ohio. 

The spread of pragmatism in education and the writings of James 
and Dewey brought on t'le progressive education movement. The 
emphasis in the mathematics curriculum shifted from the formal, 
if not rigorous, approach to an emphasis on the soci? significant 
us^'S of the subject. General mathematics courses sprea . widely and 
much of mathematics below the high school became a kind of non- 
structured, socialized, business arithmetic. Consumer mathematics 
prevailed at the expense of any interrelated mathematical structure. 
Curriculum building via job analysis almost spelled the death knell 
of mathematics as a unified system with interrelated concepts and 
themes. 

It was in this context, although delayed by World War II, that 
the present reform movement in mathematics took place. The move- 
ment was well underway before the first Sputnik occurred in 1957. 
The movement marked a swing once again from the empirical, in- 
formal, non-rigorous applied mathematics of social and business 
arithmetic to a systematic treatment of mathematical concepts. 

The modem reformers were motivated by several strong consid- 
erations: 

1. In the nation s classrooms of 1930 there was very little sign 
of the considerable body of new mathematics developed Mnce 
1900 or its uses. 

2. Because of its lack of mathematical organization and unity, 
the older curriculum failed to develop real understanding 
and the ability to extend and apply knowledge. 

3. Often what was being taught was not significant mathemati- 
cally and was distorted, if not incorrect. 

Now what are the major features of the new curricula in mathe- 
matics which have evolved since 1930? Probably the most impor- 
tant feature is the approach, rather than the content. Over 90 per 
cent of the older mafliematics has been retained but is developed 
differently. Some less significant topics have been discarded. 
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Much is involved in the new approach. In the first place, tb** 
pedagogy of mathematics has shifted from telling to asking, the 
from lecture to heuristic techniques, from memorizing to discov^,.^ 
from rote repetition to probing. The student has been changed from 
a passive to an active learner. The new approach also implies a 
new attitude toward mathematics. The student is led to experience 
a little of the thought processes of a mathematician. Mathematics 
is approached as an intriguing game of inquiry and discovery to 
be enjoyed for its own sake. The new approach generates what has 
been described as a mathematical atmosphere in the classroom. 

It is significant that, although the new curricula in mathematics 
with regards to content stress mathematical correctness, mathemati- 
cally significant topics, and the unity of mathematics through inter- 
related concepts, the new curricula \vith regards to pedagogy or 
approach did not return to the formal, expository procedures of the 
past but rather turned to new and stimulating heuristic processes 
which have been described above. 

What have the foregoing points meant for the new elementary 
school programs? For contrast, let me first describe the traditional 
eler^entary program: 

ames, numerals and meaning of numbers introduced for- 
mally, usually by rote counting. 

2. Addition and subtraction facts developed by flash cards. 
Numerous, formal exercises. 

3. Simple story problems introduced. 

4. Story problems gradually become more complex and apply 
chiefly to grocery and domestic arithmetic. Later, business 
problems are introduced. 

5. Fractions, decimals and per cent introduced with formal re- 
lations between them memorized. 

6. Denominate numbers formally taught. Some work in men- 
suration with formulas memorized. 

7. Very little on space perception, geometry, properties of num- 
bers, algebra. Procedures memorized and then practiced. 

An up-to-date modem elementary program could be described as 
follows: 

1. The meanings of cardinal and ordinal numbers developed by 
a variety of manipulative structural materials. Names and 
numerals emerge in this context. 
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2. Number facts of the fundamental operations are developed 
by children experimenting with materials and symbols. 

3. Simple story problems are developed by children using phys- 
ical contexts. 

4. Fundamental relations among numbers, such as the com- 
mutative and associative principles of addition and multipli- 
cation are developed intuitively from physical situations. 

3. Properties of the real number system are gradually developed 
by the children. Concepts of zero, one, closure, multiplica- 
tive and additive inverses, commutative, associative and dis- 
tributive laws are developed gradually by the children. 

6. The number line is used quite early to develop the meaning 
of the operations, understanding of the fundamental proper- 
ties of Ae real numbers, and in the invention of new num- 
bers, such as the fractions and negatives. 

7. Inequality ideas are developed with the equality. 

8. Children explore such properties of numbers as odds and 
evens, clocks and calendar arithmetic, number arrays, series, 
etc. 

9. Numeration systems tu other bases are investigated in order 
to increase understanding of the decimal system. 

10. Number frames and open number sentences are considered. 
Children grow accustomed here to horizontal impressions 
and literal symbols for numbers such as: What are all num- 
bers N, such that N + 3 is greater than 7? 

11. Space perception and informal geometry are developed. 

12. Intuitive concepts of measurement are developed and most 
of the ordinary mensurational formulas are developed by the 
children. 

13. ^ 'ide experience in both asldrg and answering questions is 
provided. 

14 Elementary statistical ideas are developed. 

15. Graphs evolve from work with the number line. Interpreta- 
tion as well as construction of graphs is emphasized. 

16. The ideas of fraction, decimal, per cent are developed by 
children and used in varieties of problems. 

17. The use of good logic by teachers in everyday situations for 
emulation by children. The use of true and false state^iionts 
and open statements, 

18. The notion of sets applied to number. 
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19. The later development of the algorisms for computation 
using the properties of real numbers described in #5 above. 

It can be seen that the modem program described above leads 
very naturally into the mathematics of the secondary school. The 
properties in number 5 above become the basic l^.ws of elementary 
algebra. The gradual filling in of the real number line (see 6 above) 
with fractions, positive and negative numbers, and irrational num- 
bers and later followed by the imaginary number axis perpendicular 
to the real number axis leads to the establishment of the complex 
number plane. With the complex number plane, all roots of equa- 
tions in the algebra of the secondary school can be represented. 

Most of the major curricular projects agree with the spirit of the 
description above far more than they differ. Some of these projects 
for the elementary school are: School Mathematics Study Group, 
Greater Cleveland Mathematics Program, University of Illinois 
Arithmetic Project, Madison Project, Stanford Project, and the Min- 
nesota Elen^^ntary Curriculum Project. Some of these projects for 
the secondary school are: School Mathematics Study Group, Greater 
Cleveland Mathematics Program, Unive'-sity of Illinois Committee 
on School Mathematics, Boston College Mathematics Institute, The 
Syracuse University-Webster College Madison Project, University 
of Maryland Project, The Ontario Mathematics Commission, and 
the Developmental Project in Secondary Mathematics at Southern 
Illinois University. 

Some people have criticized these programs as swinging too far 
toward abstract mathematics. Although it is true that the new pro- 
giams look upon mathematics as the queen, rather than the hand- 
maiden of the sciences, there is no valid reason for claiming that 
they are too abstract even for children. In the **queen" approach, 
the organizing basis of the new cturicula is the mathematical struc- 
ture of mathematics with its unity and interrelatedness. This is 
opposed to the unorganized social and business mathematics 
curricula of tlie past. However, the "queen" approach is not neces- 
sarily extremely rigorous or abstract. In fact, the new pedagogy of 
mathematics stresses carefully designed physical devices as aids 
to concept formation, discovery methods, appeal to intuition, and 
inductive approaches. Formal deductive methods of proof are 
evolved later. The new symbolism in contemporary mathematics 
programs is intuitively developed and is no more excessive or 
abstract than the formal algorisms of the past, or the three cases 
of per cent. 

ERIC 
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Criticis have said that the "queen" approach, in contrast to the 
'Tiandmaiden" approach, neglect^ the important role of mathematics 
in the sciences. It is true that in the new programs, mathematical 
concepts are not necessarily introduced in the context of genuine 
scientific problems. However, in at least three of the contemporary 
secondary programs a strong eflfort is being made to deal vrfth the 
important scientific applications of mathematics. For example, in 
the Boston College Program, the text "Sets, Operations, and Pat- 
terns" will have the companion textbook "Concepts of the Physical 
Universe for Man the Observer, Man the Experimenter, and Man 
the Mathematician." The School Mathematics Study Group has 
developed the following supplementary titles: "Mathematics and 
Living Things," "Badioactive Decay," "Applied Mathematics in the 
High School," and "Mathematical Methods in Science." SMSG has 
also produced revised texts for slower students. 

The evaluation of new mathematics programs has been handi- 
capped, as is often the case in curricular evaluation, by the lack 
of appropriate tests and measuring instruments. Too often, new 
programs are compared with old programs entirely on the basis of 
old tests. New tests based on the newer objectives need also to be 
designed and used in curricular evaluation. Then evaluation can 
show how successful the new programs are, not only on some of 
the older objectives, but also on a whole range of newer objectives. 
This type of new test development is a feature of the National Longi- 
tudinal Study of Mathematical Ability which is the evaluation part 
of the School Mathematics Study Group. 

The existence of a variety of new and older curricula in mathe- 
matics makes the task of developing nationally used mathematics 
tests such as the College Entrance Examination Board tests and the 
Cooperative Mathematics Tests much more difficult. Every effort is 
made to reduce the element of bias for or against any one program. 
Technical language peculiar to one or a few programs is avoided. 
Great effort is made to develop a mathematically correct, but neutral 
language. Special topics not widely taught are avoided. Questions 
are written, reviewed by a representative committee, and then pre- 
tested before they are used. By these means, the fairest tests pos- 
sible are developed. 

In this talk, I have hoped to give you some historical perspective 
on change in mathematics education and to give you an up-to-date 
look at the essential characteristics of the current mathematics pro- 
grams In the schools. 



■ Chairman Reller: I think it is appropriate that we should move from 
the field of mathematics to that of foreign languages for 1 suppose that 
if we want to consider the swing of the pendulum, certainly we wiU find 
it in both of these areas. ^ , , 

A very noted Dutch educator was in my office a couple of years ago. 
I had previously worked with her in Amsterdam and had known her tor 

" ^AUhe toi'she was with me she said that it was typical for Americans 
to "swing with the pendulum" and wish to do now in foreign language, 
what the Dutch have never contemplated doing, though for a hundred 
years they have taught foreign languages rather well they Uj°«ght. I m 
not arguing for or against these "swings." I'm not exammmg Ae question 
of how we can best achieve change or whether it is in accord with the 
inevitability of gradualness or wide swings. 

This moVninfo to look with us at some aspects of this problem we have 
a man who did his undergraduate work in a university m China, who 
Ln studted here in journalism and finally took the D«tor of Philosophy 
See in Chinese Literature and Thought at Stanford University He lias 
haS a varied experience as a reporter, as an instnictor a Ae Monterey 
Lancmage School, as a Research Assistant and Lecturer at Stairford Uni- 
versftv, as a Professor of Humanities and Foreign Languages at San Fran- 
cisco State and at present is Chairman of the Department of Foreign 
Languages there. He has also been acHve in the consideration of the 
foreign language program in the State of Cahforma. I niight say tha^ in 
addition to the matters which I have already mentioned, his major interest 
is in poetry. I think this is a very good base from w,-«ch to be operating 
as weMoolZat some of these problems. It gives me a ©eat deal of pleasure 
then to present at this time Dr. Hsu who will discuss Articulation Prob- 
lems in California Foreign Language Instruction." Dr. Hsu. 



Articulation Problems in California 
Foreign Language Instruction 

KAI-YU HSU 

The Problems Observed 

In American college education, gone are the days when a Socaratic 
peripatetic could spend long years with a handful of disciples to 
mold their pattern of intellectual growth from adolescence through 
adulthood. Cold, impersonal, stainless steel fagades have replaced 
the cloistered covered walks. Microphone and television screen are 
here to stay between the teacher and the student. And yet, no 
amount of nostalgia for a vanished golden past is going to do any 
good. It was exactly eighteen years ago that I went to work for a 
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newspaper in Chinatown in San Francisco where the residents were 
in the habit of picking up their telephones and asking for their 
parties by first names, sometimes even nicknames. And the operator 
would be able to connect them without faU. With much pain and 
persuasion, I instigated a newspaper campaign to change to the 
dialing system, all for the sake of efficiency and progress. With this 
memory still fresh, what could I say to a dean who argued for the 
installation of an IBM system whereby every one of the 16,000 stu- 
dents on my campus was to receive a number and henceforth be 
identified only by that number. 

The number of students turned away from the door of San Fran- 
cisco State College alone at present stands at over 7,000 every 
semester. Within the foreseeable future this figure will grow. Along 
with this growth will continue the trend of depersonalization in edu- 
cation. The entire process of education has been, of necessity, more 
closely paralleling an assembly line. The frightful prospect of a 
teacher seeing himself stationed at only one point on this long con- 
veyor belt to apply his screwdriver or paintbrush only for a brief 
moment before the article moves on to receive another finishing 
touch from another teacher, has already become a reality. In this 
situation, small wonder that we educators are compelled to address 
ourselves to the question of articulation. If we cannot change the 
pattern of progress altogether, at least we ought to try to insure that 
the conveyor belt is a continuous one with no break on the loop. 

Unfortunately when we turn to look for the continuous loop, we 
cannot even finchthe conveyor belt. Certainly, if there was a belt, 
very few students^succeeded in seeing it The foreign language pro- 
grams in this country at the various levels each started in their own 
ways for their own purposes. There has been very little, if any, 
meeting of the minds about what the programs at each level ought 
to be doing, and there has been certainly no more agreement even 
among the schools at the same level. At the risk of stretching the 
same metaphor to the breaking point, the various parts of what 
ought to be a machine have been cast in various shops with no 
over-all design as a guideline. Naturally, there could be no conveyor 
belt to assemble them. 

California's foreign language programs have shared the nation's 
common ills in foreign language instruction, and these ills have 
been compounded with certain local maladies. Scanning the situa- 
tion of today in this state, what major problems do we discover? 
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The first and foremost difficulty with which we are confronted is 
the question of objectives. Each institution has its own tradition, 
and tradition is usually bound with very tough hide. Most of the 
collegiate foreign language programs are still almost exclusively 
literature-oriented. A student graduating from one of these programs 
is expected to know a lot about literature, its style and analysis and 
criticism. He is expected to be very conversant with the names 
and dates and major works in a literature which he must be able 
to articulate in English. Whether or not the graduate is able to par- 
ticipate in a sophisticated discussion of literature in the foreign 
language is of very small concern. Consequendy, a student can do 
very well and graduate with a major in a foreign language if he is 
weD read in comparative literature and in secondary, comparative 
interpretation and criticism of the literature in the English language. 
This is particularly true of the relatively less taught languages in 
this country. As recently as last October, a well-known scholar in a 
foreign language field expressed himself at a high level conference 
at Princeton University, that he would not wish to see the literature 
in a foreign language sacrificed which he believed was bound to 
happen if a course in that literature had to be taught in that lan- 
guage. Indeed, I find the line very hard to draw between the 
advanced courses in foreign language and literature and those in 
comparative literature, although they fall under two different de- 
partments in most institutions. Some professors with a serious desire 
to enable their students to learn the foreign language a little more, 
insist on dealing with the original texts closely, but in so doing they 
turn their classes into mere translation exercises and sessions of 
exiguous study. They may deal, with painstaking care, with, the 
lexical items, syntactical features, or even etymology and the phi- 
losophy of the language of the passage at hand, but they fail to 
treat it as literature. These professors expect their students to have 
the kind of preparation suitable for tfus type of operation. 

To prepare students for this kind of college experience, the sec- 
ondary school programs used to give tiheir students much grammar 
analysis and reading. As the seccadary school teachers were pre- 
pared through the same kind of college programs, they were ready 
only to do the same kind of job. If nothing had happened to disturb 
this world of blissful self-content, the tradition would have been 
perpetuated forever. But the world did change, and the Russian 
Sputnik was shot up in 1957. With a sudden upsurge of demand for 
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foreign language students to be able to use the language in live 
situations in the modem world, the secondary schools being less 
tradition bound reacted more sensitively to the needs of the time 
to modify their approaches to foreign language teaching. The situa- 
tion is paraUel to the introduction of modem living into a primitive 
society where the simple innovation of covering up the woman's 
bosom actually created a total disintegration of the entire cultural 
fabric. Now pandemonium has broken loose. Those high schools 
which have adopted the so-caUed modem approach to foreign 
language teaching by stressing speech and fluency, have found their 
graduates at a disadvantage when they have to compete with the 
other students trained in the traditional method in a tradition-bound 
college program. 

This was just the beginning of the articulation problem. Imme- 
diately connected with this diversity of objectives has come the 
problem of placing the high school students in coUege programs. 
The high schools in California have been writing franticaUy to the 
colleges to inquire how their students would be butchered upon 
their entry into the college program. Should their students, having 
completed a four or five yeai sequence of study in a foreign lan- 
guage, be permitted to enroll in the third or fourth semester of the 
same language at the college level, or should they skip the lower 
division language classes altogether to embark upon upper division 
work immediately? If they were permitted to do so, and if they 
proved themselves to be capable of following the advanced level 
courses in literature at the coUege level, should these same students 
be permitted to claim coUege level credit for what they have covered? 
If tfiey were able to do so and able to ckim full credit, would this 
mean that a good high school student, having completed four years 
or more of training in a foreign language, could literally graduate 
from a college >2»tfmtegree in that language in two years time or 
less becau§e/fhe regular coUege major requires only four years of 
study? On the other hand, if a high school graduate with four years 
or more of a foreign language cannot join the advanced classes at 
the coUege level, is he permitted to repeat the lower division coUege 
level courses? If so, should it be considered remedial work for him 
and consequendy earning no credit? To give no credit for this kind 
of apparently remedial work would do injustice to those students 
whose language study at the high school level is inadequate not 
due to their own fault but rather the fault of the program itself. 



ERIC 



16 KAI-YU HSU 

And yet, giving credit for this kind of study would encourage many 
students to try to place themselves purposely in a language class 
in college lower than their level of accomplishment in order to earn 
easy recognition. Furthermore, there is a question about whether 
or not language skills are properly part of the secondary curriculum, 
hence deserving no college credit 

A second related problem occurs in the junior college program. 
The junior college in California is legally the thirteenth and four- 
teenth grades. It is authorized to offer only lower division classes. 
If a high school foreign language student, having successfuOy com- 
pleted four years of training, enrolls in a junior college, what can 
he study? Theoretically, the junior college has nothing to offer to 
this student. Thus, unless the function of the junior college curricu- 
lum is redefined, and the artificiality of the line between the so-called 
upper-division and lower-division courses is removed, such a high 
school student will have to suspend his study of the same foreign 
language, or merely take some courses which would only enable 
him to maintain his level of achievement but not to advance it any 
further. There is a serious objection on the part of the four-year 
colleges to permit the junior colleges to offer more advanced classes 
which are substitutes for the advanced work in the four-year college 
curriculum. The fear has been generated by the possibility that 
ouce this happens, the four-year college would lose control over 
the substance and coverage of a foreign language major. If the 
problem is not solved a student having learned a foreign language 
well at the high school level will be forced to suspend it for at 
least two years while he is going through his junior college career. 

We always hope that the students will do better than the teacher 
in the end, but these outstanding ones do not amount to over five 
per cent of the total in each student generation. The rest, ^ per cent, 
turn out to be, as usual, various degrees of imitation of tlieir teachers. 
It has already become a habit, perhaps rightly so, that any search 
for the roots of fordgn language instruction problems always winds 
up with the teacher to blame. Being teachers ourselves, we ate ready 
to take the blame. What kind of foreign language teachers are we 
and what kind of teachers have we been training for the schools? 
There are those very sophisticated in a foreign literature but 
extremely rusty in their use of the foreign language. There are 
others with a native speaker s facility in the foreign language but 
rustic in handling ideas and abstractions in the literature. This is 
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the wide spectrum that forms the broad base of the triangle. The 
apex of the triangle is occupied by the precious few who combine 
the two extremes. Recognition of the fact that all the future teachers 
of foreign languages must first be given solid training in the lan- 
guages before they can be expected to acquire a useful command 
of and a valid insight into the literature has been coming along 
quite slowly. As recently as last month, when the chairman of the 
foreign language departments of the California State College System 
met for the first time in four years, the battle line was still taut 
between the worshippers of literature and the advocates for linguis- 
tics. The gap may be slowly closing, but there is still a gap wide 
enough to trap many students. 

Even if the teacher training programs somehow got together and 
standardized their terminology with regard to course requirements, 
course descriptions, and number of units, what each program can 
achieve still remains unknown because the instructional effective- 
ness and the pedagogical approach of these seemingly equivalent 
courses wfll remain unknown. Turning our attention to the unknown, 
those of us concerned with the identification of professional stand- 
ards realize that we need to learn first what is actually happening 
in the field of foreign language instruction. This means a continuous 
process of gathering data and analyzing and interpreting the infor- 
mation gathered. The heavy flow of correspondence across my desk 
testifies to the general hunger for such usable information. The 
elementary school asks the high school how it should arrange its 
foreign language sequence in order to feed smoothly into the sec- 
ondary level. The high school asks the college how much credit its 
students could expect when they enter the college curriculum. The 
junior college wants to know if certain courses they are offering 
have equivalence in the college. The college wonders if it is asking 
too much or too little of its foreign language majors when the uni- 
versity program seems to be asking for something else. How can 
these questions be satisfactorily answered without some reliable 
instrument of measurement for both the quantity and the quality 
of the various parts of these programs? 

The Causes 

Now that the foreign language teacher is personally under the 
spotlight, and he writhes and reels more noticeably under the public 
eye, it does not mean that he is the only educator having loads and 



ERIC 



18 KAI-YU HSU 



loads of trouble. For if we begin at the beginning, the most funda- 
mental cause for much of the foreign language teachers woes lies 
in the disproportionate recognition given to what good education 
means and the price it exacts. It is not realistic to expect the num- 
ber of contact hours in foreign language instruction to mean any- 
thing specific when every high school teacher ia this country is 
required to teach at least five hours a day. The difference in effec- 
tiveness between the first hour and the last hour of oral drill on a 
sc} ool day needs to be but is not recognized. The size of a class 
ranges from a minimum of 25 to 45, when 15 is the optimum figure 
accepted in the profession. Wherever the third or fourth year class 
falls below the figure of 25, the classes are combined to present 
literally multi-level problems to test the ingenuity of the teacher. 
With all these limitations and variables, how is it possible to expect 
a foreign language class to achieve certain definite levels of uniform 
accomplishment after a certain number of years? Once explained, 
the public agrees that it is unreasonable to expect this, but it pro- 
ceeds to condemn the teacher as if he were the only one responsible 
for all the troubles. Thus, untQ the American society is ready to 
pay the proper price for quality education, any effort to articulate 
among the levels of programs can at best be a temporary relief of 
the symptoms, not to be taken as a substitute for a real cure. 

Besides the logistical problem, there is a still tougher one, the 
philosophical problem. Just exactly what do we expect education to 
do to a student? The humanist has one answer, and the empirical 
scientist has another. Foreign language instruction can equip the 
student with a useful skill; it can also give him a broader intellectual 
horizon without necessarily giving him a usable skill. It is noble to 
say that every program should be planned to serve both ends, but 
both ends do not always find a common ground in every course in 
a program. Consequently, there are varying stresses and slants, 
emphases and balance of emphases. Most of the quarrel over how 
any foreign language course should be taught principally stems from 
these different philosophical points of departure. Take the culture 
course, for example. Every foreign language teacher agrees that 
such a course is essential and should be included in every foreign 
language program. But that is where the agreement ends. Beyond 
it, some teachers want to make the course an intellectual history, 
while others prefer to treat it as a sociological analysis, and still 
others see no other duty in this course than teaching the students 
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relatively advanced conversation and composition. When we see a 
course entitled The Culture of France" or "The Civilization of the 
Spanish Speaking World," we have no way of knowing what goes 
on, and we have no way of equating it with another course under 
a similar title in a different program. 

Ultimately, it is the society's value system that pronounces the 
final verdict to tell good education from bad education. Ideally edu- 
cation is to produce doers who are at the same time thinkers, and 
foreign language instruction is to produce competent users of the 
foreign language who are at the same time literary critics and cul- 
tural historians, the ideal being always somewhere in the future, 
while we having to live in the present, have to face the choice 
between a doer and a thinker, between a foreign language user and 
a literary critic of that foreign literature. Time was when only a 
literary Levite was credited for knowing a foreign language, and 
even now the best teacher of foreign languages still has a hard time 
to win that measure of scholarly recognition which is his due. 
Viewed in this light, the articulation problem becomes one of 
society's searching in its own conscience to make up its mind about 
ascribing equal values to different kinds of excellence. 

These are the general but fundamental causes of the problems. 
In California a few local factors have aggravated the situation. 

There is something to be said about the wild west spirit in Cali- 
fornia. It has been said that California, with all its Spanish past, a 
persistent Oriental invasion, a northern coastline dotted with Russian 
forts, and a fishing fleet manned by Sicilian speaking settlers, really 
has no tradition. If it has any intellectual tradition, it is the tradi- 
tion of defying tradition, always pulling up its tent pegs in search 
of a greener pasture. The development of California education cer- 
tainly reflects much of this spirit. There is only a very small minority 
who strives to imitate the New England aristocracy, but most Cali- 
fomians recognize the utility of the speech of the Mexican braceros 
and Chinatown ''Chinks." The fact that the state could pass a law 
requiring foreign language instruction at the elementary school 
level reflects more of the local sentiment than a rational recognition 
of national interest. This local factor prompted California to react 
sensitively to the new wave of foreign language instruction started 
a bare decade ago to stress the mastery of the modem speech. 
Acceptance of this trend at the popular level is mirrored in many 
public policy statements. The California Association of Secondary 
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School Administrators, for example, recently distributed an oflScial 
memorandum to the State Colleges, warning that foreign language 
teachers not trained in the audio-lingual method are out of date. 
However the term audio-lingual may have been understood or mis- 
understood, the public image of a successful foreign language instruc- 
tional program in California is that it teaches students to speak the 
language fluently. This is to the good. However, before the public 
demand for results is translated into concrete logistical support for 
an educational program, the pressure is on the school administrator 
and on the teacher. The result is what I described awhile ago: the 
classroom teacher is being asked to perform better than he did 
before, but under the same physical restrictions and limitations. So 
there is a law, decreeing that all the elementary schools must teach 
a foreign language by such and such a date. So the school adminis- 
trator reacted by resorting to all sorts of make-shift arrangements 
because no appropriation has been written into the legislative man- 
date. Hiere are extreme cases where a school district makes a quick 
check throu^ its personnel files to spot two or three teachers with 
several units of Spanish on their records, and assigns them to begin 
teaching the sixth graders Spanish the following semester. One joke 
that has been circulating around is about a teacher who has been 
assigned to teach Spanish simply because his name sounds Spanish, 
even though he was bom in Milwaukee, and California is the most 
Spanish-cultured area he has ever visited in his lifetime. 

The tendency in California of rushing toward Spanish, especially 
at the elementary school level, is certainly not helping the articula- 
tion problem. In the first place, there could be more poorly taught 
foreign language classes in this state during the next five years 
than have been observed anywhere else on e^th, and these pro- 
grams simply cannot be properly articulated. In the second place, 
the same tendency generates pressure on foreign language programs 
other than Spanish, truncating their growth in some cases and elbow- 
ing them out of business in others. Today, in California foreign 
language instruction, 99.5 per cent of the programs at the elemen- 
tary school level are Spanish.* What the rest one-half of one per cent 
consists of I don't know, and it doesn't seem to matter. Over 60 per 
cent of secondary school foreign language instruction is in Spanish. 
About 30 per cent at the secondary level is in French. The rest of 
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California secondary school foreign language programs are divided 
among German, Latin, Russian, Italian, Clunese, Japanese and 
Hebrew with a small fraction for each.* Clearly, there is no articu- 
lation problem for these relatively less taught languages in Cali- 
fornia, because there are no programs vo articulate. 

Some Corrective Steps 

If these sound altogether too gloomy, I must declare that in 
the sunny state of California every patch of rainy cloud has a silver 
lining. Certainly I am not the only educator who views the situation 
with alarm. A number of corrective and preventive steps have been 
taken. 

There has been erosion in the walk of non-communication. The 
two foreign language professional organizations in the state have 
brought together mainly the State College, Junior College and sec- 
ondary school teachers of foreign languages. Foreign language 
teachers of elementary schools, a newly arisen group, are beginning 
to work closely with the other three segments of the education 
system just mentioned. Three years ago, the state-wide articulation 
conference appointed a Stanuing Committee to work on foreign 
language problems. On this committee, all five segments of the 
schooling system in California are represented, thus involving the 
university level as well. The machinery in the Department of Edu- 
cation of the state government was reorganized last winter to 
facilitate a vertical articulation. The elementary, secondary, and 
junior college foreign language specialists there are now able to 
talk to each other, learning from each other about their respective 
special problems. Through such intense, criss-cross, inter-communi- 
cation, both vertically and horizontally, th. articulation problems 
I outlined at the beginning of my discussion are being scrutinized. 
And through the continued dialogue and colloquia, some pattern of 
events to come seems to be emerging* 

One of the most encouraging signs is the gradually emerging 
general recognition that the desirable level of achievement of a pro- 
gram can be both qualified and quantified. To be sure the job is 
not an easy one, but it can be done. There is a concerted eflFort being 
exerted at the various levels of foreign language instruction to do 
precisely this job* Toward this end, the efiForts are being supported 

• In 1963, total secondary Foreign Language enrollments were 386,579, of 
which 209,390 w^e Spanish, 103>319 vere French* 
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by the accomplishments of institutions such as the Educational Test- 
ing Service. The California State Department of Education has 
already adopted the Modem Language Association tests for teacher 
certification under certain conditions. It is very likely that the same 
standard of certification may be used in the near future for all 
teaching credential candidates regardless of how they have acquired 
their preparation. It will go a long way toward the establishment 
of certain valid and uniform standards in the profession, which in 
turn will facilitate the articulation of programs at various levels. 
For placement and for measuring achievement, many institutions 
are currently experimenting with the MLA Cooperative Tests. Most 
of the graduate schools have adopted the ETS foreign language 
reading tests. The foreign language professional organizations and 
the articulatoiy agencies in the state are currently working diligently 
to build up popular support for adoption of the tests. 

Althou^ not enough progress has been made yet along this line, 
there has been some serious eflfort to differentiate the types of 
foreign language programs for different objectives. One such impor- 
tant distinction to be recognized is the line between an ideal foreign 
language program for college-bound students and another for stu- 
dents whose high school education will be terminal. When these 
different types of programs are established in California the com- 
plexity of articulation problems will be considerably reduced. 

Besides adopting standard tests and differentiating types of pro- 
grams, a third development is afoot in the Foreign Language field 
to help improve articulation. The colleges and universities are plan- 
ning "transfer** courses for freshman or transfer students whose 
foreign language background has to be adjusted before they can be 
fitted into the college sequence. The courses may be offered as 
remedial, or special summer study programs. Whichever way they 
are offered, they will ease the articulation difficulty. 

I am quite aware of the unavoidable /oaction to these commend- 
able efforts to set standards for articulation. In my ear have been 
ringing the lout' '.ries against standardization, regimentation, and 
centralization. There will continue to be such protests, principally 
from the college and university quarters, which are essential to the 
continued free exploration of ideas and methods in education. How- 
ever, the very fact that there is need for inter-communication toward 
a properly articulated sequence of language programs points up the 
diversity of policy and practice in the language field in aU segments, 



ARTICULATION . . . FOREIGN LANGUAGE 23 



particularly among the university and college campuses. Diversity 
is fine but chaos is intolerable. The university and college programs 
are supposed to set the tune to which the entire academic com- 
munity dances. But since at present each campus has its own musi- 
cal scale, there is no common tune to which the high schools could 
dance. The result is inebriate kinetics, a nightmare to every con- 
scientious foreign language teacher. It is true that we have to main- 
tain some intellectual monasteries where a few Socratic peripatetics 
can continue to conduct their independent, nonconformist search 
for the unknown. But we cannot afford to ignce the intellectual 
welfare of the 400,000 college students in this state who deserve the 
best we can afford. And certainly the best we can offer them is 
not a chaotic inarticulated foreign language sequence. I am con- 
vinced that among our fellow educators, the professional conference 
goers (the fratemizers who jump on every popular bandwagon and 
echo every education fad) and the isolationists who say "public be 
damned," are really a very small minority. The majority of us do 
not believe that intellectual excellence and educational standards 
are incompatible. Consequently, well articulated foreign language 
programs in California are not visionary but realistic goals which 
can be reached. 



■ Chairman Reller: I suppose that one of the real dangers is that we 
may well adopt tests long before we work through the philosophical 
questions or determine what the objectives of instruction are. 

As I think of this problem I am impressed at how culture-bound, in a 
sense, all societies are. I recall some seven or eight years ago when a 
citizens committee in France-and the idea of a Citizens Committee was 
rather revolutionary in France, may I say-took a look at and reported on 
some educational problems. When it brought in the report urging the 
discontinuance of the baccalaureate examinations it stirred up a great 
deal of turmoil. I recall being interviewed by the representative of one 
of tiie Paris newspapers. He wanted to know if it was true, as had been 
said in this Citizens Committee Report, that in the United States there 
were no external examinations regularly employed. He just couldnt 
understand how any school system could function at all under these 
circumstances. Why do teachers teach? Who keeps them doing anything, 
constructively, about working toward standards? This thought was veiy, 
very threatening to him. I think if he had heard Dr. Hsu, he might have 
concluded that at least in the foreign language area we were not achiev- 
ing veiy much. 

This is certainly a veiy good background and indicates the size of the 
problem thai confronts us, for» I repeat, it will be much easier to arrive 
at an examination than it will be to really work through the philosophical 
bases of action in such a field. And I suspect that this kind of rather 
quick solution may, in the long run, have difficulties just about a^ great 
as previous solutions. And so diflPerent societies, each one cultu-c-bound, 
more or less, move back and forth with reference to this educational prob- 
lem. I think one of the things that we need to recognize about this educa- 
tional revolution is that it is not confined to our society. It's going on all 
over the world, and can give us some measure of security, perhaps, as 
we struggle with these extremely difficult problems. 

For our third presentation this morning we are going to turn to an 
issue, a subject that grows out of these two earlier presentations, a con- 
sideration of the problems of "External Examinations and Continuity." 

We are fortunate in having with us for this presentation an individual 
who is director of the Advanced Placement Program of the College En- 
trance Examination Board. Before joining the staff of this group he was 
Associate Dean of the School of General Studies at Columbia University 
for a period of nine years and Associate Director of Universitj^ Admis- 
sions and Lecturer in English at Columbia. I might say that his under- 
graduate work was done in Columbia College where he both played 
varsity football and was elected to Phi Beta Kappa. This should qualify 
him to handle this extremely difficult subject. Mr. Arbolino. 
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and Continuity 

JACK N. ARBOLINO 

The College Entrance Examination Board has established a Coun- 
cil on College-level Examinations. This action is the result of long 
interest and, more recently, intensive study of a set of special prob- 
lems. It is also the formal beginning of what may be the most excit- 
ing project the Board has ever attempted. Simply stated, the plan 
calls for the development of a national system of placement and 
credit by examination. The concept is vaulting and success is not 
assured but who can deny how dubious a distinction is a record 
of success so heavy in itself that it c'ncumbers flights of aspiration. 
In short, the Trustees of the Board, who evidently suffer very few 
constrictions of the heart, seem to agree that the Board's hope should 
exceed its grasp or what's our charter for? 

We aspire through the Council to do no less than improve access 
to higher education, to facilitate mobility and individual develop- 
ment, and to provide better utilization of human resources. In 
short, we hope to use external examinations to aid continuity, or, in 
more specific terms, the Council will try to meet some distinct but 
clearly related needs: the needs of able secondary school students, 
currently a concern of the Advanced Placement Program; and the 
needs of transfer students and off-campus, or unaffiliated students. 

If the satisfaction of these needs is not worth nearly all we can 
give, the mere existence of them seemed at least a compelling 
rationale for study and action. There has been study, the Trustees 
have acted, and I would like today to tell you some of the back- 
ground of the project and some of the tentative plans for it. 

For many years we have been interested in and sensitive to prob- 
lems of transition and placement at levels other than the traditional 
one between school and college. In 1949 we developed **Intermediate 
Tests," a testing program for transfer at the end of the sophomore 
year. It failed. Part of the original battery, the aptitude section, was 
continued in 1951 as the College Transfer Test. This program also 
failed to gain wide acceptance and was abandoned in 1955. 
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In 1960 a member of the staflf wrote (forgive me now for here, 
alas, I stoop to quote myself); 

We must scrutinize credentiak and devise tests that will yield 
enough infoimation to enable us to certify, place, and divide into 
sections all students in all subjects from grades 11 to 14 and 
maybe beyond. The development of such a testing program is a 
frightening thought and a tremendous task, but I know of no 
more important job in American education. Admissions and 
placement must be refined where they have been rudimentary, 
protean where they have been procrustean.^ 

Background 

The College Entrance Examination Board may be the agency 
best suited to try to develop a national system of placement and 
credit by examination. 

If the Board, wifli help, is equal to the task it is because of its 
power, scope, and governmental structure. The last makes accept- 
ance meaningful and execution possible. Experience, a reputation 
for competence and objectivity, a record of success, and even the 
tinsel of prestige— all these are characteristics or trappings of the 
Board. However, only our members give us life and only tfieir con- 
certed r-^tions give us power In the past they have given much of 
both. ^<ow we need them both again. 

There have been other tries and other investigations, some limited 
and some inconclusive. This is no derogation of earlier attempts; 
none has been quite like the one we contemplate now and many 
will contribute much to it The others come from places as close or 
far as Buffalo or Redbrick. Many of them show the work of 
imaginative and experienced educators. It seems clear that none will 
yield all that will be needed, for the Coimcil's program will be 
national, it will consist of more than just a battery of tests, and it 
will be as adaptable to the changing pattern of preparation as we 
can make it. 

It should be stated that the Board did not seek this responsibility; 
neither has it been thrust upon us. To trace and identify the proclivi- 
ties and pressures that combine to make us move would take a wise 
and devious historian— and more time than we can spend. A balance 
of pulls and pushes meets the moment so that now the effort, mani- 
fest and worthwhile, is being made. 

One such moment can be told-it may suffice. 

The Committee on Institutional Cooperation-The Big Ten plus 
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the University of Chicago-with encouragement from the Carnegie 
Corporation had been engaged in developing a plan for work with 
iinafiiHated students. On June 8, 1964 the CIC was visited in Chicago 
and the Board's interest in the project was discussed. In a letter 
to the President of the Board dated Jiuie 26, 1964, and signed by 
the Executive Director, the following action by CIC was reported: 

The CIC (1) voted to dismiss its own Committee on Testing; 
(2) further expressed CIC appreciation to the Carnegie Corpora- 
tion for the confidence placed in CIC in suggesting that it un- 
dertake the testing project and expressed regret that it has been 
unable to implement the original testing program; and (3) noted 
its feeling that the problem represented in the previous CIC 
proposal is still of utmost importance and its belief that a com- 
petent and experienced organization such as CEEB is an appro- 
priate body to devote its attention to the question. 

The Need 

About this project it has been said, if there is no need, lead It 
has also been stated, it would be calamitous not to do it. At the 
other extreme, prudent observers who look at it plainly say, ''Whereas 
the clienteler or **Stay out, it's a swamp " 

From The Ring and the Booky and Rashomon, or the current Hol- 
lywood version which does it with cowboys, we learn there has 
never been a better device for drama than that of two men who 
look at the same thing but see it diflFerently; but that*s no way to 
run a testing organization. Statistics, opinion, and rhetoric abound 
and with eloquence enchanters can lend forecasts the semblance of 
reality. Nevertheless, the best thing to be said is that the demand 
for services is difiicult to predict The need may be something else. 

Preliminary Plans 

In barest outline the plan calls for the establishment of a Council 
on College-level Examinations which will sponsor an examination 
program. The Council will be supported by two committees: the 
Advanced Placement Committee and the Special Student Com- 
mittee. 

In formulating general plans those concerned have felt that the 
Board may be looked to for program activity at the college testing 
level because: (1) it represents a means through which the interest 
of college and university faculties in examinations of appropriate 
standards can be expressed; (2) as an agency of the colleges it is 
disinterested as well as external; (3) it should be capable of pro- 
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viding and identifying examinations of integrity. The important 
function of identification must be understood and emphasized. It is 
not now a part of any Board program. The Board has with ETS 
conceived, developed, administered, scored and graded, analyzed, 
and described various tests and examinations. But it has not sanc- 
tioned for its use any but its own. It does not identify and endorse 
for action with any of its constituency a test in no way indigenous 
to Princeton or New York. This is now proposed. It should be part 
of the Council's task to identify and use the best tests it can find 
no matter what their origin. This provisioji is to take account not 
only of such activities as the New Yoik State College Proficiency 
Examinations, ETS's Comprehensive College Tests, and others, but 
by extending endorsement and by allowing for the exercise of free 
selection it can assure the Council of the best choice of tests and 
broadest coverage of subjects. Of course unnecessary duplication of 
efiFort, in the form of needlessly overlapping examinations, could 
result in a triumph of confusion that would leave chaos institution- 
alized if not enthroned. Duplication should be avoided but diversity 
which genuinely parallels the course offerings of American colleges 
and universities should be sought. 

The amalgamation we are seeking, however, must not proceed 
to the point where the needs of able students in the secondary 
school are indistinguishable from those of the special student. There 
are unique circumstances in both divisions which, in the case of 
the Advanced Placement students, must be preserved and which, 
in the case of the others, must be considered. The virtues of ad* 
vanced placement should not be cloistered but there are singulari- 
ties which arise l.irgely in the form of supporting activities and, 
typically do not bear directly on the nature of the examinations. For 
this reason it seems necessary to provide for two separate program 
activities: Advanced Placement Program and what we may call 
Special, both under flie general guidance and advice of the Council. 

The Council might have seven main functions: (1) to formulate 
standards that examinations should meet in order to be appropriate 
for credit, placement, or transfer; (2) to apply these standards and 
identify the examinations wherever they may be; (3) to sponsor new 
examinations where there are none or where those that exist are 
inadequate; (4) to advise the President on all activities supporting 
the examination program, specifically includii ^ services in guidance, 
administraHon, evaluation and research and i Jtmation; (5) to work 
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to eflFect within colleges and universities that acceptance of the Pro- 
gram which will assure successful candidates proper recognition 
and reward; (6) to advise the President on the coordination and the 
work of the two subordmate programs; and (7) to engage in various 
experiments in credit by examination. 

The members of such a group must be good enough not only to 
do but decorate their work. It has been visualized essentially as an 
academic body made up of faculty members and academic deans 
from both two and four year colleges. 

Much can be learned from the Advanced Placement experience; 
to cite just one example, the use of the annual subject conferences 
to gain school and college acceptance of the Program. It seems 
obvious, too, that one small heresy of the Advanced Placement 
experience— recognize achievement no matter where or when it 
occurs-should serve as canon for the Council. 

The Advanced Placement Program has analogues at both levels; 
Council and program committees. It knows some of the paths and 
some of the pitfdls of the examining phase as well as the promo- 
tional phase. And if only because it precedes the Council in the 
chronology of Board programs it seems sensible to look to it for 
all the help it can give, and at the same time make sure that not 
only are its activities in no way diminished but actually furthered 
and enhanced. 

Preliminary plans call for the provision of general guidance serv- 
ices. These services would include help for transfer students as well 
as unaffiliated students. The Council must study how the University 
of London meets this problem. It should consider the use of uni- 
versity counseling centers; public libraries; various service centers 
and the numerous national educational agencies. 

Informational publications will be essential, so will a transcript 
service. If one of the basic functions of the Council is to affect 
within colleges and universities that acceptance of the program 
which will assure proper recognition, the plan to gain university 
cooperation should be recognized at the outset as vitally important. 
It is all right to let truth be in the field but it sometimes needs help. 
This is not to say there should be a great campaign to convert facul- 
ties by bombast and pamphlet. The thing itself, not the description 
of it, is aU that is ultimately important and although the program 
may well lend itself to flights of rhetoric and spurts of legitimate 
publicity, nothing is more odious to many an irmer academic man. 



ERIC 



30 JACK N. ARBOLINO 



If then the Council sees university cooperation as vitally important, 
it must plan carefully to enlist bellwethers. You name them. The 
Council must make the program good enough to engender in 
teachers and administrators tlie kind of interest that vdll make 
them feel a personal sense of responsibility for its welfare. It must 
get from the Advanced Placement Committee its own appraisal of 
the similar effort in that area, where it failed and where it suc- 
ceeded. It must demonstrate and always maintain the program's 
absolute quality. It must pretest, evaluate, and provide opportunity 
for review. It must be as careful and as fine as the Advanced Place- 
ment Program. 

Finally it is planned constantly to cultivate a commitment to this 
program that will encourage the academic community at large to 
contribute its talents, wisdom, and effort. This can be done if the 
program is the best thing it can be and, paradoxically, if those in 
charge of it never claim too much for testing. 

Some of the Problems 

Can the interests of the junior colleges be furthered through the 
Council and its program as envisioned thus far? If the Council can- 
not help it certainly should not hurt. 

To what extent can institutional self-evaluation be accomplished 
throu^ this program? 

Can an < "camination series serve multiple needs: admission, place- 
ment, and evaluation? 

How far and how fast should the Council go to serve the variety 
of objectives of the junior colleges? 

There will be no shortage of questions and subjects for research 
and study; for instance, within a university where, really, does the 
authority on credit reside? The Advanced Placement practitioners 
have been seeking that answer for ten years and it remains elusive. 

How does the Council effect within colleges that willing suspen- 
sion of disbelief or red tape that will result in honest appraisal? 

How does it, concerned as it will be with examinations and credit, 
keep primary emphasis not on testing but on learning? 

How does it find that best balance between essay and objective 
material? 

How does it take its stand on ^^influencing the curriculum?" One 
observer recommends that from this project should come a bank of 
examinations on which faculty members may draw for standardized 
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end-of-course examinations. Another describes the Board's current 
posture as neurotic: the SAT, and the achievement tests dealing 
with aptitude and achievement, do not influence but measure, or 
do they? The Advanced Placement Program clearly tries to "influ- 
ence." Will the other examinations of the Council only recognize 
and measure? 

And does the Council, concerned as it must be with rigor and 
quality, keep standards reasonable? 

A more subtle area of concern is the American concept, as dis- 
tinct, perhaps from the British, of the worth of **the course." This 
problem is similar though not as simple as the one the Advanced 
Placement Program faces when it encounters a professor who feels 
that no combination of secondary school course, high school teacher, 
and national examination can equal his course. It will be more dif- 
ficult to encounter a college teacher who, accepting the premise of 
the external examination, still retains the personal set of criteria 
he would have injected into the hypothetical course had it been his. 

What can the Council do to ease residence requirements? 

Is there a palliative for the small college which may, by full 
participation, see a decrease in tuition income? 

There are undoubtedly many other problems still unseen. It seems 
clear too that the project will require the Board's very best effort 
and even getting it, may falter. If this were to happen the only 
solace might lie in the belief that it is better to falter or even fail 
than to shirk a responsibility. 

The Trustees were unanimous in their recommendation that this 
work go forward. The Council has been appointed and an Executive 
Director has been named. The submission of proposals to founda- 
tions for developmental work has been authorized, and periodic 
reports will be submitted by the President during the next three 
years. 

The Trustees were enthusiastic as well as unanimous. Neverthe- 
less everyone concerned with the project realized that the Council 
will face some formidable issues. We accept that. Joseph Wood 
Krutch has said when we assay testing or any form of human en- 
deavor the proper observation we should make is not Pavlov's 
"How like a dog," but Hamlets ''How like a god." I mean no blas- 
phemy, but basically that is what this is all about-aspiration, and 
access, and continuity. 

Can we bring it off? Well, if it isn't easy it might be impossible- 
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it's just that difiBcult. But it will work and when it does the College 
Board ^vill be contributing to an important reconciliation in Ameri- 
can education. Increasingly we seem to seek universal education 
and quality too. Maybe a clear intention powerfully shared and 
executed with insight, initiative, and faith can effect the equality of 
opportunity and excellence we always promise. We are forever con- 
fronted with bristling alternatives: the liberal or the useful, the 
ivy or the junior, past or present, God or man, and the lists are 
endless and the rival claims are legion. But how do they bristle 
and how are they rival? The Council can begin by taking some 
things out of opposition; it can recognize the strength in diversity. 
It would be asking too much to see one day the adherents of social 
responsibility and those of academic detachment truly reconciled— 
that may not even be important-but those who must worry about 
the whole of education, and the Board is among them, had better 
see the worth of both camps, for truly to treasure learning is not 
only to transmit it, but to recognize it and see it used. 



1 Jack N. Arbolino, "Establishing and Maintaining Better Curricular Articu- 
lation Between Secondary and Higher Education," Current Issues in Higher 
Education (Association for Higher Education, NEA, 1961), pp. 149-50. 



■ Chaibman Reller: I had the feeling this morning that we had recessed 
on a note which recognized the enormous diflBculty of the problem which 
confronts us. It reminded me a little of a story I heard just a week ago 
at a Seminar dealing with problems of communication and the difficulty 
of really winning in these difficult matters. Two young men had been out 
of college about five years when they met. One had married and the other 
had not. The one who had married said to the other, "Why haven't you 
married?" "Well," he said, "I've wanted to and Tve made many eflforts 
but when I found a girl whom I liked I would take her home and my 
mother would disapprove, so I didn't marry. But, one night at a dance 
hall I met a girl who was just the girl. I knew she was the girl because 
she looked like my mother, she talked like my mother, even her nose had 
a little up-turn just like my mother, so I thought that now I really would 
win. So, I introduced myself and eventually took her home and then my 
father didn t like her." So, our problems are difficult 

This afternoon we are going to turn to a couple of additional matters, 
the first of which deals with the matter of "What's in a Norm-Testing 
What We Really Teach." For this presentation we have an individual who 
has been active in teaching and administrative work, largely in Wash- 
ington and Oregon, has been active in curriculum and planning activities 
and who for the last five years has been deeply involved in the Metro 
Testing Project. He has served on the Statistics Committee and helped 
to plan and implement the selection and construction of this new battery. 
It gives me a great deal of pleasure to present Mr. Keezel who will dis- 
cuss this problem. Mr. KeezeL 

What's in a Norm —Testing What 

We Really Teach 

£. L. KEEZEL 

Before we talk testing Yd better bring my biases out in the open. 
Achievement tests should measure learning— what supposedly takes 
place when we teach. They should be accurate and appropriate* 
When I look at a test score as a teacher, l*m often not too sure just 
what that score means. What did the test measure— how well— how 
good a job did the student do to earn this score— compared to whom? 
What reasons do I have to consider this score along with my own 
judgment of a student*s accomplishment? 

We need an objective yardstick to measure teaching, one which 
involves reasonably manageable variables* We need to be able to 
agree that a test measures what we want to teach* Just any stand- 
ardized test will not do this. National percentile scores in my area 
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are too high or too varied to make the tests very useful for guidance 
and curriculum work, though they might be good for publicity. 
(This results, of course, from our superior staff and students, though 
some people might mistakenly consider this to be a bias.) 

It is customary to cite authority for ideas or procedures, so I 
should like to offer two sources— both brief, both well done and 
both useful in practice: (The authors of the first may or may not 
be recognized here, but they enjoy considerable prestige elsewhere.) 
Henry Chauncey and John E. Dobbin, Testing, Its Place in Educa- 
tion Today and Kenneth F. McLaughlin, Interpretation of Test 
Results. I wish to offer a few quotations from these works, but this 
is a project report on the Portland-Tri-County Testing Project, not 
a research paper. What we offer is not necessarily ideal— philosophi- 
cally or experimentally— but we believe it to be sound, and it has 
been working for us. 

Chauncey and Dobbin provide this to help us develop a rationale: 

'The final step in building an achievement test Is one that 
teachers find practically impossible to accomplish with the tests 
they make themselves. This is the process of testing large num- 
bers of students so that there can be reference groups with 
which to compare an individual's score and give it meaning."* 

"Because of the practical diflSculties that stand in the way of 
the teacher who seeks to obtain norms on his test outside his 
own class, almost all teacher-made tests are limited in their 
interpretation to class norms.*** 

"Comparison data, test performance information about a lot 
of other similar students, are especially important when the per- 
formance of a particular student or class is to be interpreted."* 

Chauncey and Dobbin go on to point out: 

"Comparison with class data has the most meaning because it 
shows how well the individual has done his test Job in compari- 
son with other students who have had the same teacher, text- 
books, and curriculum. Comparison with system data lines him 
up in a still larger but somewhat more heterogeneous group. 
The comparison becomes more diflBcult to interpret as the group 
with which the student is compared grows larger and less 
homogeneous. The so-called national norms are the least useful 
of all because one so seldom knows Jiist what kinds of students 
are in the comparison groups.** * 



• Henry Chauncey and John E. Dobbin, Testing, Us Place in Education 
Today, New York, Harper and Row, 1963, pp. 47, 49. 
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McLaughlin tells us: 

"Local nonns are established on flie basis of the same stu- 
dents within a smaller area-a state, county, city or school. If 
norms are constructed for the same students for several tests at 
the same time, th^ will be comparable. Also, these local norms 
compare the student with his peers in his own community." * 

McLaughlin also says: 

"Since achievement tests can be helpful to teachers, it is im- 
portant that such tests be selected with care. Before a final 
choice is made, test content should be compared with the 
appropriate curriculum to determine whether the items included 
are covered in the local program and would be fair to the 
students."* 

Basically, the Portland Metropolitan Test Project, or more affec- 
tlonately-Metro Testing-has dealt with these problems: 
Norming and Standard Score Reporting for the Metro Area 
Test selection and construction to conform with the area's 
curnoulum 

The Metro Project began in 1959 to attack these problems. While 
I do not wish to use up too much time with background material, 
a few credits and comments are in order. Many people have been 
involved, but three have been the real prime movers since the in- 
ception of the project: Dr. George Ingebo of Portland, the late 
Bernice Tucker of Multnomah County, and Warren Adams of 
Clackamas County. 

In 1959, almost all the schools in our area were using standardized 
tests-generally the same ones. However, many of us found fliat the 
list scores and confidential score reports from the publishers gave us 
some pretty negatively skewed distributions. In my area 60 per cent 
of our graduates start college. To add to this already unrealistic 
presstire for higher education is not really fair or honest Page after 
page of 99th percentile composite scores, with the bulk of each 
class at or near the level where the student should plan to go to 
college— according to the publisher— can make things a bit sticky. 
For good measure, it was an expensive program. 

Some schools found it necessary to adopt their own devices to 

• Kenneth F.. McLaughlin, Interpretation of Test Results, U.S. Office of 
Education, Department of Health, Education and Welfare, U.S. Government 
Printing Office, Washington, 1964, pp. 49, 14. 
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explain these difFerences and make scores more meaningful. For 
instance, Beaverton Schools adopted the technique of marking each 
profile reporting card with the school average for each grade and 
test. This help d parents, sHidents, teachers and counselors to com- 
pare the student to his peers, rather than a mythical average. We 
did this with a small frame, plastic cutouts, a fistful of red ball- 
points and student help. This was only partially satisfactory, and 
reaction varied widely with the perceptual set of each individual. 
There was a need for more realistic non.. ng. 

The more-or-less obvious answer was to develop local norms and 
scores and back these up with interpretive materials— information, 
audio-visuals and research. This, Metro set out to do. 

The logistics of this project have been formidable. Metro Testing 
now involves 40 high schools and encompasses test selection; data 
processing; d^ c;ioping materials and schedules for purchasing, 
administering, scoring, norming, reporting and interpreting the tests; 
accounting; evaluation and revision; and, recently, coming full- 
circle to the selection of new tests. 

For further background information I recomn^end the pamphlet 
Developing Local NormSy published under the direction of Mrs. 
Tucker and available frc^ Portland School District No. 1, Mult- 
nomah County and Clacxamas County Intermediate Education 
Districts. 

In this pamphlet there is a description of the development of the 
Metro entity— a cooperative working relationship among more schools 
than one can customarily get to agree about anything. 

Other things involved with the project which we can only men- 
tion now are the problems of reporting, adapting all the mechanics 
of processing to the machines available and on order, cutting costs, 
developing uniform procc ^lu-es, and maintaining not only full Metro 
participation but adequate professional standajds and a reasonable 
degree of harmony. The key reason for the progress made has been 
the eflFort invested by the personnel involved. (As an aside-any area 
contemplating such a project should make sure of the commitment 
of the people involved and be willing to grant or assign some time, 
or they might just as well forget it. A project of this sort will not 
"just happen.** There is too much involved and there are too many 
steps to be taken.) 

One of the first tasks was to assure confidentiality of information, 
and there was an immediate and clear understanding by all partici- 
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pating schools that it would be a breach of professional ethics for 
any school to publicize its standing in terms of school averages com- 
pared with metropolitan area averages. No attempt was to be made 
to compare schools or evaluate teachers, and as far as I know, there 
has never been a breach of these ethics. 

We selected the tests then in use by most schools, but it was 
necessary to agree on forms and date^ f testing, scoring and report- 
ing, materials and procedures, and to make interpretative assistance 
available, so that each school was equipped to handle the data. We 
agreed on testing in the fall of grades 9 and 11 and on the use of 
standard scores rather than percentfles or grade levels. Raw scores 
were all fed into one data processing center-ours happened to be 
Service Bureau— for computation of means, deviations and standard 
errors, and production of standard scores, print-outs, profile cards 
and press labels. 

The details are myriad and since the more important ones come 
up later with the development of the new battery 111 skip them 
here. 

One separate item that began with the original ITED program 
deserves mention. Though it is barely started, it will undoubtedly 
be more important in the new battery— the use of scores as multiple 
correlation predictors for future educational success. The original 
formulae were under study by high school and college counselors, 
but this has been sidetracked in the bustle of developing and 
norming a new battery. 

This was the beginning of the project. However, there were some 
problems. Our tests began to age, and not always gracefully. Mate- 
rial copyrighted in 1957 received less than acclaim from teachers 
in light of curriculum developments in recent years. There were 
some v?^ul criticisms leveled at the tests on grounds that they were 
neither current nor appropriate to the curriculum. We had to choose 
a new battery or forget the whole thing. However, coming up 
with an achievement test battery is easier said than selected. 

When we try to measure achievement, our obligation is to the 
people who have the most at stake— students, teachers and parents. 
They must be able to accept the results at face value, not have to 
do a lot of compensating for flaws in the test or statistical niJ^nipu- 
lations. Test questions need to be teacher-selected or teacher writ- 
ten to accurately and adequately cover the subject matter involved. 
They must present a wide-enough range of diflSculty for the students 
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tested that scores will distinguish those who have learned from those 
who have not. The distribution of scores should be normal enough 
that we may have confidence in the interpretation we gp^e to dif- 
ferences between scores. This confidence must be borne out by being 
able to make better predictions with the tests than without them. 

Before a test battery could be developed for Metro use, several 
considerations had to be met: The selection process had to involve 
a broad and representative group of teachers; valid selection and/or 
construction procedures had to be observed; sampling and analysis 
had to indicate likelihood of .success. 

Perhaps we should again be a little surprised that 40 odd schools 
(and you can take that any way you want) should be able to agree 
on a selection procedure and carry it out without major incident. 
A chart plus a little descriptive material can probably best illustrate 
what was accomplished in the various subject matter areas. 

Six teacher committees in each subject matter area were estab- 
lished, one each at the 9th and 11th grade levels in each of three 
locations in the Metro area. These committees analyzed each item 
of each available test on the basis of content validity, thought 
process required in the answer— remembering, understanding and/or 
thinking— appropriateness of material, level and construction. 

The results were passed on to screening committees in each sub- 
ject area. These screening committees were composed of teachers, 
department heads, supervisors and consultants. The recommenda- 
tions of these committees varied widely. Language arts found ap- 
propriate tests for reading and composition at both levels, though 
their choices represented variations from the publisher's recom- 
mendations. Madi selected a test for each level, conforming to one 
publishers recommendations but not the others. Science recom- 
mended tests for 9th and 11th grades of one publishers sequence, 
but not at the level recommended by the publisher. Social studies 
could not find any tests which they felt were adequate for either 
level, and requested authority to construct their own, thereby com- 
mitting themselves to an even more diflBcult task. 

Shown in the chart is the diflBculty which beset the first science 
choices. In the sampling process, these tests suffered an attack of 
statisticus erraticus and had to be retired in favor of close second 
choices, equal in content validity, but much better behaved* These 
are the tests which became part of the 1964 battery. (Basically, the 
science people selected tests well above publisher's recommenda- 
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tions-difflcult enough that the results just were not reJiable.) Other 
Metro committees involved with all stages of the project include: 
steering, statistics, communication and interpretation. 
Once the tests were selected, the sampling procedure was car- 
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ried out in the spring of 1964. Tests were allocated to Metro schools 
in a random distribution. Enough students were selected individu- 
ally from flie schools' rolls to give a 10 per cent sample. The tests 
were scored; means, deviations and standard errors were computed. 
(It was in this stage that the first science tests had to be replaced.) 
Incidentally we used three experimental test times in sampling and 
obtained a good deal of information on test-taking behavior, though 
we are really just beginning to study it. 

It was also necessary to prepare administrative manuals and pro- 
cedures, to develop a coding system for the entire Mitro area which 
was compatible with the pupil accounting practices of member 
schools (some of whom use data processing systems-of various 
types— and some who do not.) New cards, labels and materials had 
to be prepared for reporting and interpretation. Reporting materials 
were standard for the Metro area, but each school district handled 
its own interpretive information program, though these were influ- 
enced by Metro representatives. 

Following the fall administration and scoring, distributions, means, 
deviations and standard scores were prepared. The whole process 
really went very smoothly, though certainly not without hitches. 
(During this year we re-leamed-among other things-that a slight 
increase in the complexity of answer sheet directions for teachers 
and students can result in a substantially higher rate of error in 
marking, and this slowed reporting.) 

There are several things in progress now which we hope will be 
accomplished before next year in addition to the completion of tibe 
social studies test. We are developing more refined reporting and 
interpreting procedures and looking for ways to make answer sheets 
more nearly foolproof— we hope they will be pre-identified before 
this fall. Standard error calculation and item analysis are in process. 
Our goal, of course, is to be able to cut test time without losing 
range and discrimination. We are almost certain we can close the 
time gap between testing and reporting. We are also recommend- 
ing a change in the structure of the Metro group this year to ask 
each district to assign a Metro representative who will be allotted 
time for Metro participation, and who can officially speak for his 
district on most matters. 

Now let's return to our social scientists. There are quite a few 
steps involved when a group commite itself to the development of 
a new test and these are shown in the following: 
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STEPS IN THE DEVELOPMENT OF THE 
SOCIAL STUDIES SKILLS TEST 

1. Social Studies Screening Committee request to construct- 
not select-a test for the new Metro battery. 

2. Administrative meeting to establish participation in Clacka- 
mas, Multnomah and Washington Counties and Portland, 

3. Meeting with ETS representatives Dana Kurfmann and 
Fred DeBruler. 

4. Appointment of a four-member (one from each area) com- 
mittee to formulate test specifications. 

5. Distribution of preliminary specifications to all school dis- 
tricts for comment and revision. 

6. Preparation of final specifications. 

7. Formation of representative teacher committees. 

8. Item-writing workshop conducted by ETS. (This workshop 
produced approximately half of the items in the three ex- 
perimental forms; the remainder were selected from ETS 
item files.) 

9. Establishing elementary and secondary curriculum to be 
covered, objectives and means of achievement. 

10. Establishing standards of construction, content, materials, 
etc. 

11. Item construction and selection from files. 

12. Critique and revisions by all participating districts. 

13. First drafts of three equivalent forms. 

14. Refinement and revision by ETS consultants. 

15. Final evaluation and approval of writing committees. (This 
step performed on the coded experimental drafts of the 
three forms.) 

16. Printing of ejmerimental forms A, B, C. 

17. Sampling and administration to 21 per cent sample (7 per 
cent form)— spring, 1965. 

18. Item analysis— difficulty and discrimination. 

19. Development of final form(s) for Metro battery. 

20. First inclusion and mass standardization-fall, 1965. 

21. Refinement and revision with other portions of the battery 
prior to 1966 administration. 

You will note from the title that this is to be a skills test. There 
are many tests on the market which measure subject matter of 
various types and this is where the social scientists disagreed. Once 
they had decided to build a test of their own, it was much easier 
to reach agreement on the content or coverage of skills test, even 
though such a test is more difficult to construct. 

They had to deviate from the format of most published tests and 
did so by setting their own goals and preparing an outline of the 
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materials and skills they wished to include. The final specifications 
agreed upon are shown below, and this balance will be maintained 
through experimental and final forms. 

PORTLAND-TRI-COUNTY SOCIAL STUDIES 
SKILLS TEST OUTLINE 
Reviiion of August 26, 1964 

Percentage 
of test 

30 L Using and Reading Printed Materials 

L Recognizing the function of a table of contents, a 
preface, a ^ossary, an index, and an appendix. 

2. Obtaining infomiation from a library card catalogue. 

3. Recognizing the types of information obtainable from 
such standard references as encyclopedias and dic- 
tionaries. 

4. Recognizing major and subordinate ideas. 

5. Making inferences. 
20 2. Using Maps and Globes 

L Understanding map projections in relation to a globe. 

2. Locating places in tenns of latitude and longitude. 

3. Estimating dist'^nces. 

4. UnderstaiKling time zones, date lines, and Great 
Circle Routes. 

5. Locating natural and man-made objects on a map. 

6. Determining directions on a map. 

7. Interpreting the map legend. 

8. Relating information from different types of maps. 

9. Making inferences. 

15 3. Using Graphic and Pictorial Materials 

L Obtaining information. 

2. Making inferences. 
35 4. Critical Thinking 

L Identifying a problem or an issue. 

2. Determining flie relevance of ideas and information 
with respect to a problem. 

3. Recognizing information needed in the solution of a 
problem. 

4. Evaluating the reliability of evidence. 

5. Distinguishing matters of fact, definition, and value. 

6. Recognizing assumptions. 

100% 

Many of the graphs, maps, etc., used in the first forms prepared 
for critique were relatively rough. Questions were coded with item 
numbers plus two digits which represented the section and subsec- 
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tion of the specifications which the item was to test— and a letter 
which showed the correct answer. These were for study purposes 
and do not appear on the tests as the student sees them. If an item 
must be replaced, it may be replaced by one measuring the same 
skill. If the three forms must be condensed into two— or even one, 
the balance shown in No. 3 may be maintained* 

In sampling these three forms this spring, we are relying on a 
seven per cent sample for each form. The sample has been arranged 
through a random selection of 8th grade classrooms in the four 
areas, and each will then be processed in a fashion similar to the 
results of the initial selections last spring, except that they must also 
be immediately subjected to item-analysis and selection. A final 
test— or tests— if there are enough good items— must be prepared— 
we hope in time to use this fall. 

We cannot deal with all the details here, but there are a few we 
should mention for the benefit of anyone else who considers develop- 
ing local tests to put more meaning into this program. It is impor- 
tant to deal with publishers who will negotiate any or all parts of 
their testing programs. (If a publisher insists on a package deal it 
will be pretty costly.) It is also important for the publisher to be will- 
ing to cooperate in the development of new tests which will meet 
area curriculum requirements— as ETS has done with us— and which 
will behave well statistically so that there can be value in the results. 
It is necessary to have adequate data processing facilities available 
at reasonable cost. 

The more valid the tests are locally, the more uses the individual 
teacher or counselor has for the results. For instance, ETS put? out 
two pamphlets titled: 'Interpreting Test Scores Realistically" and 
"Using Test Results," which the counselor or teacher can use to 
interpret and predict on his own. Rasic procedures are presented 
so that the teacher can use scattergrams and the like within his own 
classes, and this can help the teacher become more familiar with 
the tests, rather than making him suspicious of the statisticians. 
When the teacher and students are ''involved*' comparative, stand- 
ardized tests are part of the learning process. Unless they are, there 
can be questions raised about using them at all. 

This, in capsule form, is Metro Testing. It has helped us in de- 
veloping solutions to our problems, has added to our confidence 
in our test results, and to our teachers' knowledge and use of the 
information we can provide. ... It seems to work. 
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■ Chairman Reller: Thank you, Mr. Keezel for a veiy comprehensive 
review. 

We will turn now to our final presentation, this one dealing with "New 
Directions in Assessing Achievement." To make this presentation we have 
one who has had junior high school teaching experience, who has served 
on college and university teaching staffs at New York University and 
Rutgers University and who joined the Test Development Division of 
Educational Testing Service somewhat more than ten years ago. He 
became Assistant Director of Test Development, then Director, and in 
1963, Vice President. It gives me a great deal of pleasure to present 
Mr. SoIomon« 

Mr. Solomon: Thank you. Dean Reller. It strikes me that this Con- 
ference on Testing has yet to hear a joke about testing, so with your 
permission let me try. It s a story of a poor soul vAio some of us would 
describe as a pawn of fate, a child of misfortune. At any rate, wherever 
he went, whatever he did, things never went well. If he bit into an 
apple, the apple had a worm. If he put his spoon into a grapefruit, he 
was squirted in the eye. If he walked down the street, a car came by 
and splashed him with mud. 

Well, one fine day this poor soul had to take a trip to San Francisco 
and as you might expect he missed his plane and because he had to be 
at this important meeting he arranged for a chartered plane. The plane 
took off and, of course, once more, no sooner were they airborne than all 
the motors went dead. In panic, the pilot pushed a parachute into his 
arms and said, "Put it on and jump." And our poor soul put it on and 
jumped. And you know what happened. He puUed the ripcord and 
nothing happened. At this point as he was falling to earth about to meet 
his doom in desperation he turned to prayer and said, "St. Francis, St. 
Francis, save me. Save me and 111 devote the rest of my life to you.** 
And at this point, a mighty arm came out from behind a cloud, grabbed 
him in mid-air and held him. Then a magisterial voice said, ''St. Francis 
Xavier or St. Francis of Assisi?** And our poor soul thought a minute 
and then said, "St. Francis Xavier." And the hand opened and he dropped. 

The moral to the story is it never p^s to guess on two choice items, 
especially when there is a correction for guessing. 
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Testing is an old profession. Along with gunpowder, kites, and print- 
ing, testing appears to have been invented by the Chinese. But it 
has been tihe pragmatic Americans who have raised the art of test- 
ing to its present level of sophistication. Along with atomic energy, 
airplanes, and public libraries, testing may well be listed by future 
historians among the unique developments of American society. If 
so, it will reflect a recognition of the American belief in education. 
For it is because of our conviction that progress rests on the identifi- 
cation and nurture of individual talent that we have developed 
instruments that work to identify, guide, and assess individual poten- 
tial and achievement. Combine a belief in individual achievement, 
social progress, and pragmatic solutions and you have an explana- 
tion of why a Chinese invention has become an American institution. 

The close relationship between education and testing is perhaps 
best seen in the measurement of educational achievement. When 
achievement tests are constructed properly, they measure what edu- 
cators believe are the more important outcomes of instruction, and 
their use, content, and interpretation are closely related to the edu-- 
cational process. 

The past twenty years have seen a significant improvement in the 
educational relevance of achievement tests. In part, this has been 
a result of recognition that the construction of better achievement 
tests depends on better defined statements of educational objectives 
and, in turn, of the efforts that have been made to show how this 
could be done— to the benefit of teaching as well as testing. An out- 
standing example of these eflForts is the Taxonomy of Educational 
Objectives, edited by Benjamin Bloom. In part, too, the improve- 
ment in the educational validity of achievement tests has been a 
result of the efforts made to involve teachers and scholars in the 
actual planning and development of achievement tests. For example, 
for each achievement test ETS develops in a subject field, there is 
a panel of specialists in the field who plan, write, review, and 
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approve the test before it is used. In a typical year, about 600 such 
individuals participate in developing tests at ETS. 

Still another force for the improvement of achievement tesls has 
been the efforts made to improve the ways of measuring achieve- 
ment. A little over a year ago ETS published a booklet entitled 
Multiple-Choice Questions: A Close Look. Perhaps many of you 
have seen it; if not, it is available from ETS without cost As the 
introduction to the booklet states, **The purpose of this close look 
at a group of multiple-choice questions is to dispel a myth: the myth 
of the multiple-choice question as a superficial exercise. . . . Like 
other myths, this one may be based on a shadowy memory of tfie 
past, but it bears little relation to present reality.'' Although in 
the past, achievement tests— multiple-choice and essay— often tended 
to measure little more than factual recall, and although too fre- 
quently a few still do, there is now ample demonstration that this 
need not be. There are volumes of literaturfr-of which Multiple- 
Choice Questions: A Close Look is merely one-that show how the 
construction of achievement tests can be consistent Avith and sup- 
porting of important educational objectives. 

Recently, there have been several developments that show how 
the art of achievement testing continues to move in the direction of 
the construction of tests that are educationally valid. Some have been 
made possible by new technology that was not widely available 
until a few years ago. Some are old wines in new bottles, but in 
at least one case the new, improved bottle is the result of research 
studies started more than a decade ago. Still others are little more 
than a gleam in the collective eye of those who contemplate the 
future of achievement testing. 

One development or, more correctly, one series of developments 
has to do with the need to develop tests that sample as directly as 
possible the behavior that is the criterion of educational achieve- 
ment. To obtain reliable measurement and, ultimately, valid meas- 
urement as well, tests sometimes need to be structured in ways that 
seem artificial in terms of the criterion behavior to be measured. 
Usually the artificial-appearing test produces valid and useful results, 
but the ideal is a test that validly measures the criterion behavior 
because it a sample of criterion behavior-or a very reasonable 
facsimile thereof. 

A recent development that reflects this concern has been the intro- 
duction of the tab-question technique in die tests of the National 
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Board of Medical Examiners. The tab technique is useful when one 
is concerned with sequential thinking of a diagnostic problem- 
solving type. It provides a measure of behavior that involves the 
serial performance of a set of procedures, where the performance 
of one step yields information that supplies a cue for tiie selection 
of the next and subsequent steps. The name comes from the fact 
that the examinee pulk a tab each time he answers a question. The 
examinee proceeds from one question to the next on the basis of 
the information he finds under each tab. The National Boaid of Med- 
ical Examiners uses the tab technique to simulate the procedures a 
doctor would follow in diagnosing and treating an illness. To my 
knowledge the technique was first used by Robert Glazer and the 
late Dora Damrin to assess the performance of radar mechanics. 
As in golf, the best score is the lowest score; the fewer tabs lifted, 
the better the examinee's problem-solving proficiency. 

Another feature of the tab technique is that it has the potential 
for teaching as well as testing by providing immediate feedback to 
the student on the correctness or incorrectness of his response and 
by giving him the reasons for the correct response and explaining 
why other responses are incorrect. In this respect the tab test is 
related to many similar developments in the past forty years in 
testing and, more recently, in programmed instruction. 

Another recent development that represents the effort to develop 
tests fliat measure behavior as directly as possible is also an example 
of how tests need to change as the curriculum changes. In this case, 
the development has been prompted by a fundamental shift in the 
objectives of teaching modem languages. Although the audio-lingual 
method is not new, tfie present emphasis given to it in the teaching 
of modem languages represents a major curriculum reform. 

If tests are to be educationally valid, a change in instructional 
objectives requires a change in tests of instmctional outcomes. In 
this instance, a new type of test has been developed. It is a test of 
foreign-language speaking ability in which the student speaking 
into a tape recorder pronounces words, reads a passage aloud, de- 
scribes a series of pictures, or tells a story. The student^s tape is then 
graded by trained scorers using scoring standards in much the same 
way that trained readers grade essay tests. One remarkable charac- 
teristic of these speaking tests is that although they are essentially 
free-response tests and therefore susceptible to all the defects of 
such instruments, they can be scored with remarkable reliability. 
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Relatively short tests of about twenty minute duration consistently 
yield estimates of test reliability above .8 and frequently as high 
as .9. These speaking tests, which as a result of projects sponsored 
by the Modem Language Association and supported by the U.S. 
Office of Education are now available in French, Spanish, Russian, 
German, and Italian, to test both aspiring teachers and students 
from grade 7 through college, would probably not have been pos- 
sible a generation ago before the advent of language laboratories 
and mass-produced, low-cost tape recording equipment. 

If the development of speaking tests represents a relatively new 
curricular objective, another recent development represents an objec- 
tive as old as the American school itself. This concerns the ability 
to write. Although we know how to construct multiple-choice tests 
that are extremely effective in measuring students' writing ability, 
as determined by grades in English and teachers' ratings, we con- 
tinue to seek ways to reliably and validly measure writing ability 
by having students write. In the course of this se?rch, improved 
techniques for constructing and grading essay tests have been de- 
veloped, and a considerable body of research knowledge about the 
essay test has been accumulated. Now, as a result of the most recent 
research in this area, the College Board English Composition Test 
combines multiple-choice and essay questions in a test of students' 
ability to write with no less the reliability or validity of earlier exclu- 
sively multiple-choice tests. Although no single element in this 
approach is new, the complementary balancing of multiple-choice 
and essay questions and the method used for grading the essays 
(which relies on several global judgments instead of a single analyti- 
cal judgment) represent an important step toward resolving the per- 
sistent problem, particularly in English composition, of measuring 
criterion behavior in a reliable, valid, and relatively direct way. 

However, ultimate success in the improvement of tests to measure 
the important behavioral outcomes of instruction will require an- 
other type of development that is only now beginning. Speaking 
tests in French and writing tests in English are relatively obvious 
solutions to the measurement of criterion behavior. But if we shift 
to a subject such as biology, how then should we revise cur tests 
to measure better the behavior that is the criterion of achievement 
in biology? What is such behavior? What do we expect the student 
of biology to be able to do-and how do we know if he can do it? 
One solution is to do what specialists in the development of achieve- 
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ment tests now do. ThPc is, an expert committee of teachers and 
scholars specify the desired outcomes of instruction and then write 
questions that in their judgment measure these outcomes. The evi- 
dence that they have succeeded is their own judgment and that of 
their peers that they have done so, or the relationship between test 
scores and grades in the subject. Although this approach has been 
generally successful, its limitation is that the measurement of the 
criterion behavior is inferential. Another approach that we are now 
attempting in connection with the College Board Biology Achieve- 
ment Test is the production of the criterion behavior itself. This 
will involve the definition in behavioral terms of all the important 
outcomes of high school biology and the subsequent definition and 
development of situations in which the behavior may be manifested. 
These situations will include activities of all kinds— in school and 
out of school, in the laboratory and in the field. If the project is 
successful, it will provide a means for gathering direct evidence on 
the achie\'ement of each important objective of high school biology 
and for relating such evidence to the construction of paper-and- 
pencil achievement tests. Although the number of activities in the 
criterion measure would probably require weeks for any one student 
to complete, on a sampling basis it should be possible to determine 
which of the important outcomes of biology instruction are now 
being measured by the College Board Biology Achievement Test 
and which, if any, are not 

The educational relevance of achievement testing has many 
aspects. There is, for example, the problem of assessing those objec- 
tives of instruction that are concerned with how studencs think 
rather than what they think or know. In his recent research on the 
outcomes of instruction in the new high school physics course de- 
veloped by the Physical Science Study Committee, Robert Heath 
developed a cognitive preference test that measures not whether the 
student can solve each of the problems presented— all responses are 
correct— but rather what type of answer he prefers. Each response 
offers a different way of considering the problem. Is the student 
predisposed to consider the problem in terms of its theoretical or 
its practical implications? Does he prefer to think first of the prin- 
ciples involved or would he rather think first of related facts? Heath 
found that different physics curricula produced different cognitive 
preferences. For curriculum evaluation particularly, the cognitive 
preference test offers a new way to relate achievement testing to 
the objectives of instruction. 



50 ROBERT J. SOLOMON 



' Not all improvements in the validity of achievement tests require 
the development of new kinds of tests. One persistent problem that 
faces a school system that builds its curriculum with thought and 
care, and that selects with equal thoug.it and care the standardized 
achievement tests it uses, is that certain curricular objectives unique 
to that school system may be neglected or overlooked entirely in 
standardized tests. If so, the tests are less than ideal instruments 
for curricular or student evaluation. Faced by this dilemma the 
school may decide to use no standardized test at all-which it often 
cannot do easily— or to use the one that comes closest to measuring 
those objectives that it considers most important. Recently, however, 
the Portland (Oregon) public schools finding themselves in this pre- 
dicament proposed to ETS a novel solution. They asked ETS to 
help them build the kind of social studies test they wanted and 
to do so by adapting an existing social studies test. The project was 
begun last summer. With the assistance of a test development con- 
sultant from ETS, a team of social studies teacllers from Pordand 
reviewed their social studies objectives for junior high schools and 
developed content specifications for the tests they need. Then, 
analyzing ETS-published tests and a pool of additional test questions 
from ETS, the Portland teachers identified those objectives for 
which test questions existed and those for which new t Jst questions 
were needed. They decided that about 60 per cent of the questions 
could come from die ETS pool and about 40 per cent would have to 
be wiitten at that time. Since then, ^he new questions have been 
written and pretested. Within the next fe\ months, the new Pordand 
social snidies tests will be assembled. Alt .ough Portland probably 
could have carried forward the entire project successfully without 
ETS's assistance and its pool of questions, we believe that the project 
was made easier by having been done in the manner just described. 

The Pordant' project has interesting implications. First, it sug- 
gest the beginnings of a new approach to standardized achievement 
tests. For years we have been telling schools that they should select 
achievement tests that are consistent with their instructional objec- 
tives. In the future, ^my school that finds existing tests inadequate 
or inappropriate may be able to build its own. But, if every school 
developed its ow test, it would have no way of comparing itself 
to others, or its students to a national norm. We may therefore see 
the development of modular tests that would enable a school system 
to use modules that would be appropriate for it and many other 
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schools in combination with other modules that would be unique 
to that school. 

Further, if one also considers the demonstrated capabilities of 
computers, it is possible to foresee a time in the near future when 
an organization such as ETS could quickly and cheaply assemble 
tests to meet a school's specifications and to do so from a pool of 
test questions stored in its computer and classified by content, diffi- 
culty, etc. And as new questions were written and as new data were 
gathered with each question's use, the pool would become more 
comprehensive and more useful. Also, computers could make pos- 
sible the rapid and inexpensive computation of local norms and 
equivalent national norms for these custom tailored, computer- 
assembled tests. 

Looking beyond these tests that are computer-assembled and 
custom tailored for each school, the standardized achievement test 
of the future may be custom tailored fo* each student. Administered 
by means of computer equipment that would select for each student 
the next question he should try, depending on his responses to pre- 
vious questions, the test would permit each student to move forward 
at his own speed and to branch in different directions as his abOity 
and knowledge permitted. The information provided by such tests 
would make possible a genuinely individualized assessment and 
diagnosis of each student's achievement. 

I began by saying that testing is an old profession. I hope that I 
have shown that it is not a static one. Tests on tabs, tape recorders, 
and computers do suggest that the technology of achievement testing 
is changing. But the more important changes are those that improve 
our understanding of individual achievement and consequently our 
ability thereby to improve the education of the individual. 
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■ Chairman Reller: As 1 have listened to the presentations today and 
thought of the problem of documenting educational change, an area to 
which we have devoted relatively little thought, it seems to me that it 
is the whole matter of related staff development in connection with the 
problems of testing and evaluation which is important In his concluding 
statement Mr. Solomon pointed to the possibility of individual assess- 
ment. It and many of diese suggested developments call for attention to 
the question of staff development. I think it is a veiy central problem 
with which all of us must be veiy much concerned. 

In concluding I might relate just a final stoiy and in order that Mr. 
Solomon will not say that it is not related to the subject at hand, may 
I suggest that it does pertain to norms? The stoiy is one of the many 
about that great Englishman of our time, Winston Churchill. IxC was 
being presented in Albert Hall in London before a very large meeting of 
women. He was being introduced to representatives of the Women's Clubs 
and by a woman who had long been interested in the temperance move- 
ment. In the course of her presentation she pointed to the fact that not 
only had he played a very significant part in the life of England but cer- 
tainly also of the whole Western world and even beyond and would stand 
as one of the great leaders of men. 

But V :th her particular background she found it difficult not to make 
some reference to some of his other achievements and so she referred to 
the fact that he not only had made these remarkable achievements, these 
great contributions to the human race but that while in the process of 
doing it had actually consumed great quantities of brandy. And as she 
looked over the vast hall she said that it had been estimated that actually 
tihe brandy he had consumed would fill Albert Hall to at least a fifteen- 
foot level. When Churchill arose he looked at this point in the distance 
and slowly turned his head to survev the great hall. "Yes," he said, "I 
regret that the norms in our society have not permitted adequate recog- 
nition of individual differences. There is, as a result, still so much to 0,0 
and so short a time in which to do it.** 

May I just in closing express my thanks to all of you, those who were 
involved in the various roles of this Conference. Thank you veiy much. 
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